Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drupal good default Initial PR #6

Merged
merged 111 commits into from
Apr 27, 2020
Merged

Drupal good default Initial PR #6

merged 111 commits into from
Apr 27, 2020

Conversation

nikathone
Copy link
Member

@nikathone nikathone commented Feb 25, 2020

PR Testing Getting started

Clone the fork repo

git clone git@github.com:nikathone/isle-dc.git isle-dc-nikathone && cd isle-dc-nikathone

Running the containers

With this repo you can run drupal using drupal/recommended-project or islandora/drupal-project. I have added aMakefile to facilitate with the setup and running the needed commands. The steps below will download the codebase, setup data/drupal/ and run docker-compose.

  • Run make isle_codebase=drupal to create codebase with drupal/recommended-project
  • Or run make isle_codebase=islandora to create codebase with islandora/drupal-project
  • In case you have composer installed locally and run into the composer memory limit problem you might need to run COMPOSER_MEMORY_LIMIT=-1 make isle_codebase=islandora

You might need to run docker stats to check CPU usage for each container. Since AUTO_INSTALL is on it might take some time for the drupal container CPU usage to come down once it's done. So give the drupal container up to 5-10min to complete the site installation. Then visit islandora.localhost:8000 in chrome or if using any non chromium web browser add islandora.localhost in the /etc/hosts.

Bringing down the containers

  • Run make down to just bring down the container without cleaning up all the various docker assets related to the docker compose.
  • Or run make clean to delete everything from the codebase, data/drupal to all containers, images and volumes associated with docker-compose.yml
  • For a "light clean" you can also run make clean_local

For more make commands please check the Makefile.

Interacting with the container

  • docker-compose -p islandora exec drupal drush uli: to get a one time login url for admin.
  • docker-compose -p islandora exec drupal drush cr: to clear cache.

Notes

The drupal.Dockerfile still using registry.gitlab.com/nikathone/drupal-docker-good-defaults/php-nginx:latest for php and nginx base image but this will change once we have a CI to build images/nginx-php/Dockerfile.

I did disable the SOLR service.

Also it might be a good idea to move the traefik service setup in it's own docker-compose file cause ideally they should only be one traefik service per local host.

I might have missed something since I was trying to avoid to add to many changes at once.

@noahwsmith
Copy link
Contributor

Thanks for this!

A couple notes:

Same error. @nikathone are there other UID/GID variables I need to tweak?

@nikathone
Copy link
Member Author

Yes +1 for the data folder. Maybe then we can have something like data/drupal/files and data/drupal/database|mariadb. I will try to reproduce the last two bullet points and get back to you. Thanks for the initial feedback.

@noahwsmith
Copy link
Contributor

We manually cat'ed this to the end of the settings.php file https://gist.github.com/nikathone/e6eba02c6f8275f14d68d2b1d473c026

@noahwsmith
Copy link
Contributor

Thanks! initial test

drupal_1   | [26-Feb-2020 22:10:50] WARNING: [pool www] child 22 said into stderr: "NOTICE: PHP message: PDOException: SQLSTATE[HY000] [2002] No such file or directory in /var/www/app/web/core/lib/Drupal/Component/DependencyInjection/PhpArrayContainer.php on line 79 #0 /var/www/app/web/core/lib/Drupal/Core/Database/Driver/mysql/Connection.php(416): PDO->__construct('mysql:host=;por...', '', '', Array)"
drupal_1   | [26-Feb-2020 22:10:50] WARNING: [pool www] child 22 said into stderr: "#1 /var/www/app/web/core/lib/Drupal/Core/Database/Database.php(371): Drupal\Core\Database\Driver\mysql\Connection::open(Array)"
drupal_1   | [26-Feb-2020 22:10:50] WARNING: [pool www] child 22 said into stderr: "#2 /var/www/app/web/core/lib/Drupal/Core/Database/Database.php(166): Drupal\Core\Database\Database::openConnection('default', 'default')"
drupal_1   | [26-Feb-2020 22:10:50] WARNING: [pool www] child 22 said into stderr: "#3 [internal function]: Drupal\Core\Database\Database::getConnection('default')"
drupal_1   | [26-Feb-2020 22:10:50] WARNING: [pool www] child 22 said into stderr: "#4 /var/www/app/web/core/lib/Drupal/Component/DependencyInjection/PhpArrayContainer.php(79): call_user_func_array('Drupal\\Core\\Dat...', Array)"
drupal_1   | [26-Feb-2020 22:10:50] WARNING: [pool www] child 22 said into stderr: "#5 /var/www/app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(173): Drupal\Component\DependencyInjection\PhpArrayContainer->createService(Array, 'database')"
drupal_1   | [26-Feb-2020 22:10:50] WARNING: [pool www] child 22 said into stderr: "#6 /var/www/app/web/core/lib/Drupal..."
drupal_1   | - -  26/Feb/2020:22:10:50 +0000 "GET /index.php" 500
mariadb_1  | 2020-02-26 22:10:52 16 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
^CGracefully stopping... (press Ctrl+C again to force)
Stopping islandora_drupal_1   ... done
Stopping islandora_mariadb_1  ... done
Stopping islandora_traefik_1  ... done

I'll blow everything away and test again

@noahwsmith
Copy link
Contributor

I set $settings['hash_salt'] = 'asdfasdfasdfasdfasdfasdadsafdsaasdfadf';

Then when I run drupal site:install --force --no-interaction

I get

drupal@3efe5bdaee27:/var/www/app$ drupal site:install --force --no-interaction

 Starting Drupal 8 install process


 [ERROR] File system: Writable (public download method)

         Array




 [ERROR] <em class="placeholder">Notice</em>: Array to string conversion in <em
         class="placeholder">install_display_requirements()</em> (line <em class="placeholder">2272</em> of <em
         class="placeholder">core/includes/install.core.inc</em>). <pre
         class="backtrace">install_display_requirements(Array, Array) (Line: 1089)
         install_verify_requirements(Array) (Line: 702)
         install_run_task(Array, Array) (Line: 577)
         install_run_tasks(Array, NULL) (Line: 117)
         install_drupal(Object, Array) (Line: 583)
         Drupal\Console\Command\Site\InstallCommand-&gt;runInstaller(Array, &#039;default&#039;) (Line: 466)
         Drupal\Console\Command\Site\InstallCommand-&gt;execute(Object, Object) (Line: 255)
         Symfony\Component\Console\Command\Command-&gt;run(Object, Object) (Line: 1000)
         Symfony\Component\Console\Application-&gt;doRunCommand(Object, Object, Object) (Line: 255)
         Symfony\Component\Console\Application-&gt;doRun(Object, Object) (Line: 188)
         Drupal\Console\Core\Application-&gt;doRun(Object, Object) (Line: 64)
         Drupal\Console\Application-&gt;doRun(Object, Object) (Line: 148)
         Symfony\Component\Console\Application-&gt;run() (Line: 89)
         require(&#039;/var/www/app/vendor/drupal/console/bin/drupal.php&#039;) (Line: 4)
         </pre>


@noahwsmith
Copy link
Contributor

Looks like files is the problem: fixing the initial instructions

mkdir -p drupal_files/public drupal_files/private mariadb_files/public mariadb_files/private

@noahwsmith
Copy link
Contributor

Nia and I screenshared again- there's still an issue, this time with the Mariadb. He's going to work on this and revise the PR, and then I'll test again.

@nikathone
Copy link
Member Author

@noahwsmith I have been working on this script which is almost done and will add it on this PR.

It's goal is to make it easier to create and prepare the codebase , the data/drupal/files and the data/drupal/database folders.

@noahwsmith
Copy link
Contributor

Thank you! I'll stay tuned...

@nikathone
Copy link
Member Author

Ok. I things are much easier to test. I will update the pull request description accordingly.

@noahwsmith
Copy link
Contributor

@nikathone this works very nicely on first run! Thank you.

@g7morris and I will spend some more time with it tomorrow, and we'll hold some agenda time on Thursday in the 2pm Eastern call to discuss with the team.

@dannylamb
Copy link
Member

I'm consistently getting the following error when trying make isle_codebase=drupal:

[INFO] Setting up settings.isle.php to be included in settings.php
 
       An existing settings.php file couldn't be located and one is created from default.settings.php.
sed: can't read : No such file or directory
Makefile:15: recipe for target 'drupal_init' failed
make: *** [drupal_init] Error 2

I'm just flailing about though. Is there some obvious step I'm missing?

@nikathone
Copy link
Member Author

@dannylamb thanks for testing and reporting back on this issue. I will add a debug flag option to be passed onto the make actions and see if that can give us the exact error which caused sed to fail.

@g7morris
Copy link
Collaborator

g7morris commented Apr 15, 2020

@nikathone Reporting the following from testing: Everything is working great except the timezone.

Test results

Issue #1

  • Fixed. Can now change passwords and domain name.

  • Fixed. No issues with drush or docker compose when running docker-compose -p islandora exec drupal drush uli

  • Fixed. Now seeing correct port of 8000 in drush uli generated login reset link

  • Not fixed yet. Toronto is still coming up as default timezone not UTC.

Issue #2

  • Fixed. Make clean no longer reporting error when it runs.

Issue #3

  • Fixed. Can now change passwords and domain name but will review once merge takes place.

--

Remaining steps prior to merge with MVP2a & MVP3

Changes in direction prior to merge

@niakathone I hate to be a pain but can we revert the current Docker-compose service name changes of drupal_db back to using only mariadb please? I regret wasting your time here, apologies. I'm concerned with a big resource change like having three mariadbs and not having enough group architectural input as of yet. We can certainly discuss further the possibility of using multiple mariadbs as the opportunity arises post merge.

For now, I've discussed with @noahwsmith as well and we think the easiest thing we can do without having to manage 3 separate mariadbs, is to continue to use one and to add a .sql script that runs upon the first mariadb init which will create the following:

  • drupal db along with drupal_db_user and password. (Default already)
  • fcrepo db along with fcrepo_db_user and password.
  • gemini db along with gemini db user and password.

This way everything is created from the start even if users don't use fcrepo or gemini later on at least they exist and are ready to use from the first start.

I will work on this script and adapt from what we already have in MVP3. Per the https://hub.docker.com/_/mariadb section called Initializing a fresh instance, It would then be placed in the mariadb docker-entrypoint-initdb.d as a sql file which can first check upon init if all of the users and dbs exist and then creates them.

I am not sure how we would make that SQL file dynamic so I'm guessing we'll hardcode user values for now and iterate after the merge.

I think once we can clear the Solr swap and any appropriate cleanup, we can run a test merge and see what fixes are necessary.

@nikathone
Copy link
Member Author

@g7morris I just pushed a change which revert back to mariadb instead of drupal db and I am totally on board with trying one db service to serve the other services.

For the sample.env I did update the docker-compose.yml accordingly but only focused on drupal and the mariadb services. The other services are using env-file which point to .env and that doesn't make it easier to know which environment variables they really need. It could be easier for someone who understand how these services work to do the refactoring.

Also can you give a screenshot or a specific command you are using to check the timezone?

Thanks again for the tests and feedback.

@nikathone
Copy link
Member Author

One more thing once we have the sample.env and docker-compose.yml environment variables setup we should then get rid of .env, add it to .gitignore and then update the doc.

@g7morris
Copy link
Collaborator

g7morris commented Apr 15, 2020

@nikathone Thanks!

The process for checking the timezone comes up when I run docker-compose -p islandora exec drupal drush uli, copy the drush uli generated login reset link and open the url in a new browser, change the password and then scroll down to see the timezone which is stuck on Toronto. I'll rebuild in a few moments and put a screenshot up.

  • Ah yes we should put that create sample.env as a post-merge step for sure. I've added it to our Sprint 4 strategy doc and will update other lists accordingly. Thanks!

@g7morris
Copy link
Collaborator

g7morris commented Apr 15, 2020

@nikathone Re "For the sample.env I did update the docker-compose.yml accordingly but only focused on drupal and the mariadb services. The other services are using env-file which point to .env and that doesn't make it easier to know which environment variables they really need. It could be easier for someone who understand how these services work to do the refactoring."

I can work with you on that post-merge, I should be able to help corral these for the most part.

Copy link
Collaborator

@g7morris g7morris left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nikathone This looks good except on line 41 of the docker-compose.yml, should this section of the drupal service that is currently:

    depends_on:
        drupal_db:
           condition: service_healthy

actually be changed to:

    depends_on:
        mariadb:
             condition: service_healthy

@g7morris
Copy link
Collaborator

g7morris commented Apr 15, 2020

@nikathone FYI I've started the SOLR work here g7morris@98ccd36

So far so good using the official image solr:8.4.1, as I'm able to:

  • Start up the container / service without crashing or apparent errors
  • Continue to use the solr-data Docker volume this time mounted to /var/solr/data/islandora (the core)
  • Use the official image docker-entrypoint.sh along with its solr-precreate script such that when the solr service inits for the first time it creates the islandora core automatically. There is a check script to ensure no re-creation or overwrite of the core upon container restart. This means we could don't need your create-core.sh script anymore thought it was handy.
  • From what I can see in /var/solr/data/islandora/configsets
lrwxrwxrwx 1 solr solr   36 Mar 20 20:52 _default -> /opt/docker-solr/configsets/_default
lrwxrwxrwx 1 solr solr   55 Mar 20 20:52 sample_techproducts_configs -> /opt/docker-solr/configsets/sample_techproducts_configs
lrwxrwxrwx 1 solr solr   51 Mar 20 20:52 search_api_solr_8.x-3.2 -> /opt/docker-solr/configsets/search_api_solr_8.x-3.2

There is a pre-existing search_api_solr_8.x-3.2 so I don't think we need to configure Solr any further. The test I'll run tomorrow is running the Makefile and seeing what happens.

[Followup edit / question]

@nikathone Within the Makefile

I'm assuming I can simply comment out the solr_init section now e.g.

solr_init:
	./scripts/solr/create-core.sh

However for default: drupal_init up solr_init do I:

  • remove only solr_init e.g. default: drupal_init up
  • or remove up solr_init e.g default: drupal_init

Once I figure that out, I can perform the test tomorrow.

@nikathone
Copy link
Member Author

nikathone commented Apr 15, 2020

Removing

solr_init:
	./scripts/solr/create-core.sh

and editing default: drupal_init up solr_init to be default: drupal_init up should be enough.

If you want to keep the solr_init phony around while you are testing you can just edit default: drupal_init up solr_init to default: drupal_init up. This way solr_init phony won't run unless directly invoked with make solr_init.

@g7morris
Copy link
Collaborator

@nikathone Thanks for that, was able to make things work-ish.

Everything spins up correctly except Solr doesn't seem to make a core by default anymore.

When I was testing the service by itself, I could docker-compose up solr and could see a core named islandora yet when I run make the container fails to run its own entrypoint scripts which is confusing. Additionally when I docker-compose up solr the container name is the correct one e.g. isle_dc_proto2b_solr but when I run make the container name is islandora_solr_1 despite having the correct container name setting.

I also saw that both the mariadb and drupal services don't have a container_name: "${PROJECT_NAME}_mariadb" or container_name: "${PROJECT_NAME}_drupal", I'm going to test adding them.

@nikathone
Copy link
Member Author

I mean always try to avoid to hard code container name and only rely on the project and service name to figured out the container name. This way your apps won't have anything based on something which can be auto generated when deployed on live environment.

@g7morris
Copy link
Collaborator

g7morris commented Apr 15, 2020

Agreed but that then confirms there is an issue with the container and volume naming as the drupal and mariadb container names and volumes names start with islandora_ instead of the value in the .env which is PROJECT_NAME=isle_dc_proto2b

I'm okay with going the other way and striping out hard coded container name designations for all other services once we can see what is up with this.

^ Okay I'm testing commenting out all container name references now as my previous addition of all the hardcoded container names failed too.

@nikathone
Copy link
Member Author

Ok.

@g7morris
Copy link
Collaborator

g7morris commented Apr 15, 2020

Okay reporting this as an issue now:

All services in docker-compose.yml have the container_name value commented out.

The docker volumes are

volumes:
    mariadb-data:
    activemq-data:
    solr-data:

The .env settings are:

### PROJECT SETTINGS

PROJECT_NAME=isle_dc_proto2b
PROJECT_BASE_URL=idcp2b.localhost
PROJECT_SHORT_ID=idcp2b

But the outcome is:

Creating islandora_cantaloupe_1       ... done
Creating islandora_traefik_1          ... done
Creating islandora_activemq_1         ... done
Creating islandora_homarus_1          ... done
Creating islandora_mariadb_1          ... done
Creating islandora_hypercube_1        ... done
Creating islandora_crayfits_1         ... done
Creating islandora_fits_1             ... done
Creating islandora_alpaca-connector_1 ... done
Creating islandora_solr_1             ... done
Creating islandora_houdini_1          ... done
Creating islandora_drupal_1           ... done

docker ps
CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS                   PORTS                                                                    NAMES
cc7394161047        islandora_drupal                         "docker-webserver-en…"   2 minutes ago       Up 2 minutes             80/tcp, 443/tcp, 8080/tcp, 9000/tcp                                      islandora_drupal_1
2b63a74935e1        harvardlts/fitsservlet_container:1.5.0   "/usr/bin/supervisord"   3 minutes ago       Up 3 minutes             8009/tcp, 8080/tcp, 8443/tcp                                             islandora_fits_1
e63206003016        birkland/isle-alpaca                     "karaf server"           3 minutes ago       Up 3 minutes             1099/tcp, 8101/tcp, 8181/tcp, 44444/tcp                                  islandora_alpaca-connector_1
399dd17dd8d3        borndigital/isle-crayfits:mvp2-alpha     "docker-php-entrypoi…"   3 minutes ago       Up 3 minutes             80/tcp, 8000/tcp                                                         islandora_crayfits_1
b331821540cc        borndigital/isle-hypercube:mvp2-alpha    "docker-php-entrypoi…"   3 minutes ago       Up 3 minutes             80/tcp, 8000/tcp                                                         islandora_hypercube_1
6326e147891b        solr:8.4.1                               "docker-entrypoint.s…"   3 minutes ago       Up 3 minutes             0.0.0.0:8983->8983/tcp                                                   islandora_solr_1
39c7a9e93ebb        borndigital/isle-houdini:mvp2-alpha      "docker-php-entrypoi…"   3 minutes ago       Up 3 minutes             80/tcp, 8000/tcp                                                         islandora_houdini_1
0f158f8aadfe        borndigital/isle-homarus:mvp2-alpha      "docker-php-entrypoi…"   3 minutes ago       Up 3 minutes             80/tcp, 8000/tcp                                                         islandora_homarus_1
94d6b16cad13        mariadb:10.3.22                          "docker-entrypoint.s…"   3 minutes ago       Up 3 minutes (healthy)   3306/tcp                                                                 islandora_mariadb_1
0852db0ae3bb        rmohr/activemq                           "/bin/sh -c 'bin/act…"   3 minutes ago       Up 3 minutes             1883/tcp, 5672/tcp, 61613-61614/tcp, 61616/tcp, 0.0.0.0:8161->8161/tcp   islandora_activemq_1
25fc185424e7        lyrasis/cantaloupe:latest                "/build/entrypoint.sh"   3 minutes ago       Up 3 minutes (healthy)   0.0.0.0:8182->8182/tcp                                                   islandora_cantaloupe_1
a7d7b9fa4241        traefik:2.1.3                            "/entrypoint.sh --ap…"   3 minutes ago       Up 3 minutes             0.0.0.0:8000->80/tcp                                                     islandora_traefik_1

docker volume ls
local               islandora_activemq-data
local               islandora_mariadb-data
local               islandora_solr-data

I still am able to see my site at http://idcp2b.localhost:8000 which is good but still no Solr core created.

Is this partially due to line 5 in the Makefile? docker_compose_project ?= islandora

[Followup] interesting. I replaced islandora with docker_compose_project ?= isle_dc_proto2b on line 5 and got make: *** [Makefile:23: up] Error 1

I'm probably not making any more sense for the day. Thanks for the help. Let's catch up tomorrow. Excited to play with all of this.

drupal_init:
./scripts/drupal/init.sh --codebase $(isle_codebase)

solr_init:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor non-issue. Should be added to .PHONY targets.

@g7morris g7morris merged commit e638f04 into Islandora-Devops:development Apr 27, 2020
birkland added a commit to birkland/isle-dc that referenced this pull request Oct 26, 2020
Expose the database root password to Drupal environment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants